dial tcp 100.116.116.4:5672: connect: connection refused
rabbitmq-ha-39-0 1/1 Running 8 143m rabbitmq-ha-39-1 1/1 Running 4 143m rabbitmq-ha-39-2 1/1 Running 18 143m
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning BackOff 54m (x7 over 55m) kubelet Back-off restarting failed container Warning Unhealthy 19m (x379 over 149m) kubelet Readiness probe failed: dial tcp 100.126.22.67:5672: connect: connection refused Warning Unhealthy 4m50s (x311 over 146m) kubelet Liveness probe failed:
livenessProbe at all. I'd strongly recommend to use the Operator. As for the issue, logs should provide additional clues. This basically just says that the deployment defined a TCP readinessProbe that attempts to connect on 5672 but RabbitMQ is not listening on this port. Judging by the number of pod restarts, it seems like it can't start. Most likely explanations would: incorrect configuration or insufficient CPU/memory but the logs should contain additional information.
1 year ago
1 year agox-max-length-bytes/x-max-age/x-stream-max-segment-size-bytes, Is there a more flexible way? x-stream-offset and delete all messages before or after it.kubectl get servicemonitor rabbitmq -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"monitoring.coreos.com/v1","kind":"ServiceMonitor","metadata":{"annotations":{},"name":"rabbitmq","namespace":"data-samples-w02-s003"},"spec":{"endpoints":[{"interval":"15s","port":"prometheus","scheme":"http","scrapeTimeout":"14s"},{"interval":"15s","port":"prometheus-tls","scheme":"https","scrapeTimeout":"14s","tlsConfig":{"insecureSkipVerify":true}}],"namespaceSelector":{"any":true},"selector":{"matchLabels":{"app.kubernetes.io/component":"rabbitmq"}}}}
creationTimestamp: "2022-03-31T14:35:10Z"
generation: 1
name: rabbitmq
namespace: data-samples-w02-s003
resourceVersion: "18150746"
uid: d7d3d0f4-ae10-4277-99d9-f2efbb680afd
spec:
endpoints:
- interval: 15s
port: prometheus
scheme: http
scrapeTimeout: 14s
- interval: 15s
port: prometheus-tls
scheme: https
scrapeTimeout: 14s
tlsConfig:
insecureSkipVerify: true
namespaceSelector:
any: true
selector:
matchLabels:
app.kubernetes.io/component: rabbitmq
1 year ago
1 year agokubectl get pod --output=jsonpath='{.items[*].spec.containers[*].resources.limits.cpu}' --all-namespaces --selector app.kubernetes.io/component=rabbitmqkubectl get -o yaml pod -o=jsonpath='{.items[*].spec.containers[?(@.name=="rabbitmq")].resources.limits.cpu}' --all-namespaces
1 year agojsonpath in #archived-2 would limit results more efficiently than #1-archived but that's just a guess. (edited) Waiting on Support ⟶ Waiting on R&D2.0.10/3.8.27 around 2022-02-28T23:57
rabbit@c1a4ea58-65ec-43ef-80ce-8440a3c16039.rabbitmq-server.services.service-instance-e0e2b2ac-cfe8-4f29-80aa-fd1edaf6c202.bosh.log:2022-03-01 21:32:34.627 [warning] <0.21216.4327> Could not delete quorum queue 'queue 'ibus-dq-s-ENG18-TWDS_CAPTURE-*.WAVEFORM_POST' in vhost 'e0e2b2ac-cfe8-4f29-80aa-fd1edaf6c202'', not enough nodesa new leader was elected after customer recreated queueonline to reach a quorum: [{error,noproc},{timeout,{'e0e2b2ac-cfe8-4f29-80aa-fd1edaf6c202_ibus-dq-s-ENG18-TWDS_CAPTURE-*.WAVEFORM_POST','rabbit@c1a4ea58-65ec-43ef-80ce-8440a3c16039.rabbitmq-server.services.service-instance-e0e2b2ac-cfe8-4f29-80aa-fd1edaf6c202.bosh'}},{error,noproc}]. Attempting force delete.
rabbitmq-server.c1a4ea58-65ec-43ef-80ce-8440a3c16039.2022-03-02-20-27-47/rabbitmq-server/rabbit@c1a4ea58-65ec-43ef-80ce-8440a3c16039.rabbitmq-server.services.service-instance-e0e2b2ac-cfe8-4f29-80aa-fd1edaf6c202.bosh.log:2022-03-01 21:32:34.636 [info] <0.13248.0> queue 'ibus-dq-s-ENG18-TWDS_CAPTURE-*.WAVEFORM_POST' in vhost 'e0e2b2ac-cfe8-4f29-80aa-fd1edaf6c202': terminating with shutdown in state candidate
rabbitmq-server.2e61acd8-0d32-4a4c-b60b-5fcac75e86b3.2022-03-02-20-27-47/rabbitmq-server/rabbit@2e61acd8-0d32-4a4c-b60b-5fcac75e86b3.rabbitmq-server.services.service-instance-e0e2b2ac-cfe8-4f29-80aa-fd1edaf6c202.bosh.log:2022-03-02 02:45:50.973 [info] <0.30258.4636> queue 'ibus-dq-s-ENG18-TWDS_CAPTURE-*.WAVEFORM_POST' in vhost 'e0e2b2ac-cfe8-4f29-80aa-fd1edaf6c202': detected a new leader {'e0e2b2ac-cfe8-4f29-80aa-fd1edaf6c202_ibus-dq-s-ENG18-TWDS_CAPTURE-*.WAVEFORM_POST','rabbit@3e295f24-589c-43c6-921e-a19e3148346b.rabbitmq-server.services.service-instance-e0e2b2ac-cfe8-4f29-80aa-fd1edaf6c202.bosh'} in term 1I also noticed quorum related crash during the same timeline when issue was happening.2022-03-01 21:32:05 =CRASH REPORT====…
crasher:
initial call: rabbit_channel:init/1
pid: <0.24748.3935>
registered_name: []
e
Waiting on Support ⟶ Waiting on R&D
Moving this thread from Slack here, to keep a record of the work being done. Our prometheus endpoint in RabbitMQ can be configured to serve metrics over TLS. When the user choses to enable TLS in RabbitMQ, we configure the prometheus endpoint to serve over TLS as well. We also configure prom_scraper job with the property “server_name” set to the service instance BOHS DNS, as we include this name in the certificate’s Subject Alternate Name (SAN).Given the “prom_scraper_config.yml” attached, we would expect prom_scraper to connect to the SI using the BOSH DNS, however, we observe the following error repeatedly in the prom scraper log:2022/03/18 09:02:08 failed to scrape: scrape errors:Notice the metric url as
[id: rabbit@localhost, instance_id: rabbit@879479da-73df-4b72-a1d4-57aa6a947786.rabbitmq-server.sheengreen-services-subnet.service-instance-2ab41afd-f9a0-41cc-a03a-71d1c28e4d44.bosh, metric_url: https://127.0.0.1:15691/metrics]: Get "https://127.0.0.1:15691/metrics": x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs
metric_url: https://127.0.0.1:14821/metrics instead of the server name.
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year agoOpen ⟶ Waiting on R&D
1 year ago
1 year ago
1 year agorabbitmqadmin unlikely enables the same settings a browser could, this is a client/server setting mismatch. It's interesting that the browser than serves any pages at all. There may be proxies involved that add additional restrictions on POST requests (such as definition uploads) but not GET requests (that serve UI pages)
1 year ago
1 year ago# error from cf CLI verbose outputIn rabbitmq service broker log, we see following errors.
REQUEST: [2022-04-21T16:36:25-07:00]
POST /v2/service_instances?accepts_incomplete=true HTTP/1.1
Host: api.system.prod.stratus.albertsons.com
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: cf/7.4.0+e55633fed.2021-11-15 (go1.16.6; amd64 darwin)
{
"name": "test-rq4",
"service_plan_guid": "ef794f46-2e57-4c8c-8be9-db68fc4cf73c",
"space_guid": "9a050d3f-e5f7-4373-a9d3-beac2486c505"
}
RESPONSE: [2022-04-21T16:36:26-07:00]
HTTP/1.1 502 Bad Gateway
Content-Length: 207
Content-Type: application/json;charset=utf-8
Date: Thu, 21 Apr 2022 23:36:26 GMT
Server: nginx
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 78797058-c07a-4f9e-7119-7866d29c1cc2::db94e7a3-594f-4d4d-828c-87bd0db472a3
{
"code": 10001,
"description": "Service broker error: http request failed with status code: 400",
"error_code": "CF-ServiceBrokerBadResponse",
"http": {
"method": "PUT",
"status": 500
}
}
{"timestamp":"2022-04-22T01:03:23.633774328Z","level":"info","source":"rabbitmq-service-broker","message":"rabbitmq-service-broker.provision.entry","data":{"service_instance_id":"e0a64d86-add7-45d3-be95-d17ef3fa15e3","session":"91054"}}
{"timestamp":"2022-04-22T01:03:24.007928073Z","level":"error","source":"rabbitmq-service-broker","message":"rabbitmq-service-broker.provision.provision-admin-user-failed","data":{"error":"http request failed with status code: 400","session":"91054"}}
{"timestamp":"2022-04-22T01:03:24.330487029Z","level":"info","source":"rabbitmq-service-broker","message":"rabbitmq-service-broker.provision.exit","data":{"session":"91054"}}In rabbitmq-server management-ui access.log, I see vhosts have created always successfully for new instances. But "PUT /api/permissions/<vhost>/<user>" has been failing and hit HTTP 400 error code. I filtered “PUT /api/permissions“ and “PUT /api/vshost“ requests from management UI access log. (See attached recent_vhosts_create.out and recent_permissions_create.out).I notice there is particular rabbitmq-server/7aa6d1e5-cf74-43e7-b66e-1eaa239994d3 always returning 400 error code for requests "PUT /a…Open ⟶ Waiting on R&D# error from cf CLI verbose outputIn rabbitmq service broker log, we see following errors.
REQUEST: [2022-04-21T16:36:25-07:00]
POST /v2/service_instances?accepts_incomplete=true HTTP/1.1
Host: api.system.prod.stratus.albertsons.com
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: cf/7.4.0+e55633fed.2021-11-15 (go1.16.6; amd64 darwin)
{
"name": "test-rq4",
"service_plan_guid": "ef794f46-2e57-4c8c-8be9-db68fc4cf73c",
"space_guid": "9a050d3f-e5f7-4373-a9d3-beac2486c505"
}
RESPONSE: [2022-04-21T16:36:26-07:00]
HTTP/1.1 502 Bad Gateway
Content-Length: 207
Content-Type: application/json;charset=utf-8
Date: Thu, 21 Apr 2022 23:36:26 GMT
Server: nginx
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 78797058-c07a-4f9e-7119-7866d29c1cc2::db94e7a3-594f-4d4d-828c-87bd0db472a3
{
"code": 10001,
"description": "Service broker error: http request failed with status code: 400",
"error_code": "CF-ServiceBrokerBadResponse",
"http": {
"method": "PUT",
"status": 500
}
}
{"timestamp":"2022-04-22T01:03:23.633774328Z","level":"info","source":"rabbitmq-service-broker","message":"rabbitmq-service-broker.provision.entry","data":{"service_instance_id":"e0a64d86-add7-45d3-be95-d17ef3fa15e3","session":"91054"}}
{"timestamp":"2022-04-22T01:03:24.007928073Z","level":"error","source":"rabbitmq-service-broker","message":"rabbitmq-service-broker.provision.provision-admin-user-failed","data":{"error":"http request failed with status code: 400","session":"91054"}}
{"timestamp":"2022-04-22T01:03:24.330487029Z","level":"info","source":"rabbitmq-service-broker","message":"rabbitmq-service-broker.provision.exit","data":{"session":"91054"}}In rabbitmq-server management-ui access.log, I see vhosts have created always successfully for new instances. But "PUT /api/permissions/<vhost>/<user>" has been failing and hit HTTP 400 error code. I filtered “PUT /api/permissions“ and “PUT /api/vshost“ requests from management UI access log. (See attached recent_vhosts_create.out and recent_permissions_create.out).I notice there is particular rabbitmq-server/7aa6d1e5-cf74-43e7-b66e-1eaa239994d3 always returning 400 error code for requests "PUT /a…[warn] <0.619.0> Schema definition sync: no upstream settings are configured, this downstream has nowhere to connect
Ford’s PCF team gives this to their Dynatrace team to monitor their RabbitMQ SI?
monitoring tag (i.e. creating a service key in PCF), and pass those credentials to the monitoring team.
$ kubectl logs -f v1-rabbitmq-server-0 --previous
=SUPERVISOR REPORT==== 20-Apr-2022::21:36:55.320723 ===
supervisor: {local,credentials_obfuscation_sup}
errorContext: start_error
reason: {low_entropy,
[{crypto,strong_rand_bytes,1,
[{file,"crypto.erl"},{line,1054}]},
{credentials_obfuscation_svc,check,3,
[{file,"credentials_obfuscation_svc.erl"},{line,162}]},
{credentials_obfuscation_svc,init_state,0,
[{file,"credentials_obfuscation_svc.erl"},{line,133}]},
{gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},
{gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,226}]}]}
offender: [{pid,undefined},
{id,credentials_obfuscaton_svc},
{mfargs,{credentials_obfuscation_svc,start_link,[]}},
{restart_type,permanent},
{significant,false},
{shutdown,5000},
{child_type,worker}]
=CRASH REPORT==== 20-Apr-2022::21:36:55.325777 ===
crasher:
initial call: application_master:init/4
pid: <0.120.0>
registered_name: []
exception exit: {{shutdown,
{failed_to_start_child,credentials_obfuscaton_svc,
{low_entropy,
[{crypto,strong_rand_bytes,1,
[{file,"crypto.erl"},{line,1054}]},
{credentials_obfuscation_svc,check,3,
[{file,"credentials_obfuscation_svc.erl"},
{line,162}]},
{credentials_obfuscation_svc,init_state,0,
[{file,"credentials_obfuscation_svc.erl"},
{line,133}]},
{gen_server,init_it,2,
[{file,"gen_server.erl"},{line,423}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,390}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,226}]}]}}},
{credentials_obfuscation_app,start,[normal,[]]}}
in function application_master:init/4 (application_master.erl, line 142)
ancestors: [<0.119.0>]
message_queue_len: 1
messages: [{'EXIT',<0.121.0>,normal}]
links: [<0.119.0>,<0.44.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 987
stack_size: 29
reductions: 208
neighbours:
=INFO REPORT==== 20-Apr-2022::21:36:55.328884 ===
application: credentials_obfuscation
exited: {{shutdown,
{failed_to_start_child,credentials_obfuscaton_svc,
{low_entropy,
[{crypto,strong_rand_bytes,1,
[{file,"crypto.erl"},{line,1054}]},
{credentials_obfuscation_svc,check,3,
[{file,"credentials_obfuscation_svc.erl"},
{line,162}]},
{credentials_obfuscation_svc,init_state,0,
[{file,"credentials_obfuscation_svc.erl"},
{line,133}]},
{gen_server,init_it,2,
[{file,"gen_server.erl"},{line,423}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,390}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,226}]}]}}},
{credentials_obfuscation_app,start,[normal,[]]}}
type: transient
{"Kernel pid terminated",application_controller,"{application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{low_entropy,[{crypto,strong_rand_bytes,1,[{file,\"crypto.erl\"},{line,1054}]},{credentials_obfuscation_svc,check,3,[{file,\"credentials_obfuscation_svc.erl\"},{line,162}]},{credentials_obfuscation_svc,init_state,0,[{file,\"credentials_obfuscation_svc.erl\"},{line,133}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,423}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}}},{credentials_obfuscation_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{low_entropy,[{crypto,strong_rand_bytes,
Crash dump is being written to: /var/lib/rabbitmq/erl_crash.dump...done
1 year ago
1 year ago
1 year ago
1 year ago
1 year agorabbitmqctl display_disk_space_used_by_standby_replication_data to see that the downstream cluster is consuming disk space. rabbitmqctl display_disk_space_used_by_standby_replication_data command, for example:rabbitmq@downstream-rabbit-c1-server-0:~$ rabbitmqctl display_disk_space_used_by_standby_replication_data Listing disk space (in gb) used by multi-DC replication node size unit vhost rabbit@downstream-rabbit-c1-server-2.downstream-rabbit-c1-nodes.rbtmq-cluster 5.0e-4 gb test
downstream-rabbit-c1-server-2, and then I will go to the downstream-rabbit-c1-server-2 node and execute the promote command. rabbit_stream_coordinator node, right? rabbitmqctl delete_all_data_on_standby_replication_cluster , I notice it does not delete the queue and messages What does it delete in the background?Q4:Questions about SchemaReplicationreplicationPolicies.pattern like StandbyReplication? Restrict the scope of schema Replication to a certain vhost? Can you give me an example.apiVersion:rabbitmq.com/v1beta1 is different from StandbyReplication, which is apiVersion:rabbitmq.tanzu. vmware.com/v1beta1.
Looking forward to your reply Thank you! (edited) Error events: Pstandby-replication-operator-69ffbf8849-fxscc NamespaceNSbb00000015-rmq-operator-inf-01 Just now Generated from kubelet on ip-10-244-160-209.us-west-1.compute.internal 5 times in the last 0 minutes MountVolume.SetUp failed for volume "cert" : secret "standby-replication-webhook-server-cert" not found PodPstandby-replication-operator-69ffbf8849-fxscc NamespaceNSbb00000015-rmq-operator-inf-01 Just now Generated from Successfully assigned bb00000015-rmq-operator-inf-01/standby-replication-operator-69ffbf8849-fxscc to ip-10-244-160-209.us-west-1.compute.internal
1 year ago
1 year ago
6:20 AMOpen ⟶ Waiting on R&D
1 year ago
1 year ago
1 year ago
1 year agoPublish and not like the yellow from Unroutable (drop) ?rabbitmq@upstream-server-0:/$ rabbitmq-plugins enable rabbitmq_standby_replication
Enabling plugins on node rabbit@upstream-server-0.upstream-nodes.rabbitmq-system:
rabbitmq_standby_replication
Error:
{:plugins_not_found, [:rabbitmq_schema_definition_sync, :rabbitmq_schema_definition_sync_prometheus, :rabbitmq_standby_replication]}kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"
kubectl apply -f rmqcluster_upstream_1.yaml
Do I have to add the these plugin's file into rabbitmq cluster by myself or I have to create rabbitmq operator/cluster by registry image if I want to use these plugins?---apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: upstream
image in your RabbitmqCluster and it deploys the OSS image by defaultx-quorum-initial-group-size=1. Is that the case?Open ⟶ Waiting on R&D
1 year agoWaiting on Support ⟶ Waiting on R&D
1 year ago
1 year ago
1 year ago
1 year ago
1 year agoCapital Group has the request below and I provided all the relevant links. But now he has one specific question above.They want to replace TIBCO CMS with RabbitMQ. As the part of this migration, some applications are using JMS APIs. After looking at the JMS Java Client docs, I doesn't tell us how to define custom JMS attributes to use quorum queues. Can you give us the JNDI and JMS example on using quorum queues?
1 year agoRMQDestination can be given an optional map of queue attributes https://github.com/rabbitmq/rabbitmq-jms-client/blob/3930852c87e6dcfc428dd62bfd79c[…]/test/java/com/rabbitmq/integration/tests/RmqDestinationIT.java, https://rabbitmq.com/queues.html#optional-arguments, https://www.rabbitmq.com/quorum-queues.html#usage
1 year agoWe are aware of the option of using RMQQueue definition class. How do we set the same thing for JNDI destination?
rabbitmq@downstream-rabbit-c1-server-0:~$ rabbitmqctl display_disk_space_used_by_standby_replication_data Listing disk space (in gb) used by multi-DC replication node size unit vhost rabbit@downstream-rabbit-c1-server-2.downstream-rabbit-c1-nodes.rbtmq-cluster 3.0e-4 gb test rabbit@downstream-rabbit-c1-server-2.downstream-rabbit-c1-nodes.rbtmq-cluster 5.0e-4 gb test2
downstream-rabbit-c1-server-2 node and caches the synchronized message data on the disk of the node.server-2 node.kubectl delete pods downstream-rabbit-c1-server-2 --grace-period=0 --force -n rbtmq-cluster server-2 node.server-0 or server-1 and retransmit the message data from the upstream cluster to the downstream cluster.server-2 pod innovation was created, the replica process was still on it. I see through the management UI of upstream that there seems to be data retransfer from upstream to downstream.(not sure)
1 year agoTanzu RMQ to my dev VM running Rocky Linux8.5 and microk8s... I am following this guide. After installing carvel, creating and exporting a secret a have the below error - .status.usefulErrorMessage looks like low-level comms, but very strange because the VM has full connectivity. Stopping the firewalld does not make any difference. Any thoughts?[root@localhost tanzu-rabbitmq]# kapp deploy -a tanzu-rabbitmq-repo -y -f repo.yml -n tanzu-rmq Target cluster 'https://127.0.0.1:16443' (nodes: localhost.localdomain) Changes Namespace Name Kind Age Op Op st. Wait to Rs Ri tanzu-rmq tanzu-rabbitmq-repo PackageRepository 1m - - reconcile fail Reconcile failed: (message: Fetching resources: Error (see .status.usefulErrorMessage for details)) Op: 0 create, 0 delete, 0 update, 1 noop, 0 exists Wait to: 1 reconcile, 0 delete, 0 noop 12:43:44AM: ---- applying 1 changes [0/1 done] ---- 12:43:44AM: noop packagerepository/tanzu-rabbitmq-repo (packaging.carvel.dev/v1alpha1) namespace: tanzu-rmq 12:43:44AM: ---- waiting on 1 changes [0/1 done] ---- 12:43:44AM: fail: reconcile packagerepository/tanzu-rabbitmq-repo (packaging.carvel.dev/v1alpha1) namespace: tanzu-rmq 12:43:44AM: ^ Reconcile failed: (message: Fetching resources: Error (see .status.usefulErrorMessage for details)) kapp: Error: waiting on reconcile packagerepository/tanzu-rabbitmq-repo (packaging.carvel.dev/v1alpha1) namespace: tanzu-rmq: Finished unsuccessfully (Reconcile failed: (message: Fetching resources: Error (see .status.usefulErrorMessage for details))) [root@localhost tanzu-rabbitmq]# kubectl get packagerepositories/tanzu-rabbitmq-repo -n tanzu-rmq -o=jsonpath={.status.usefulErrorMessage} vendir: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents: Imgpkg: exit status 1 (stderr: imgpkg: Error: Checking if image is bundle: Fetching image: Error while preparing a transport to talk with the registry: Unable to create round tripper: Get "https://registry.tanzu.vmware.com/v2/": dial tcp: lookup registry.tanzu.vmware.com on 192.168.233.2:53: read udp 10.1.102.134:46669->192.168.233.2:53: i/o timeout )
1 year ago
1 year ago1.3.0 which will include the latest operators and 3.10.2. There is currently no plan to release another patch for 1.2 that includes 3.9.15. (edited)
1 year ago
1 year ago
1 year ago1.2 patch that includes the latest 3.9.x. We’ve already submitted the OSL request for 1.3.0 so 1.3.0 is likely going to arrive faster than a potential 1.2.x.
I reckon I also would like to ask: do we recommend running RMQ on Win? Not recommend? Neutral?
1 year ago
1 year agoStatus:
Conditions:
Last Transition Time: 2022-06-01T04:22:51Z
Message: Put "https://upstream.rabbitmq-cluster.svc:15671/api/global-parameters/schema_definition_sync_upstream": x509: certificate is valid for *.rmq.event.oocl.com, not upstream.rabbitmq-cluster.svc
Reason: FailedCreateOrUpdate
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedUpdate 61s (x21 over 41m) schema-replication-controller failed to set 'schema_definition_sync_upstream' global parameter certificate is valid for *.rmq.event.oocl.com, not upstream.rabbitmq-cluster.svc. They generated the certificate for *.rmq.even.oocl.com but are connecting to the cluster using upstream.rabbitmq-cluster.svc name. That name should be added to the certificate
1 year agomaint-23 section here - https://erlang.org/download/otp_versions_tree.html
1 year ago
1 year ago
1 year agoProject Maturity
This project is based on a mature Go client that's been around for over a decade.
We expect this client to undergo moderate breaking public API changes in 2021. Major and minor versions will be updated accordingly.
1 year ago
1 year ago
1 year agoRabbitmqCluster objects in the cluster i.e. kubectl get --no-headers --all-namespaces rmq | wc -lrabbitmq-system i.e. kubectl -n rabbitmq-system get events -o json
1 year agodebug log level?max-in-flight parameter for the tile is set to 3. This means that once you start upgrading the tile, BOSH would recreate the cluster and its VM with a batch size of 3: if you have a cluster with nodes across 3 AZ, this means that you will break existing connections. PoleEmploi faced this issue last week while deploying a new tile version. Is this a known issue? What about setting the default value to 1 for max-in-flight? Thanks for your help. /cc @Didier
1 year ago
1 year agokubectl port-forward - ?root@ubuntu:~/tanzu-rmq-deploy# kubectl get pods -n kube-prometheus -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES alertmanager-prom-kube-prometheus-stack-alertmanager-0 2/2 Running 0 8m10s 10.244.0.18 kind-control-plane <none> <none> prom-grafana-5c6db5967-rnw8b 3/3 Running 0 8m17s 10.244.0.16 kind-control-plane <none> <none> prom-kube-prometheus-stack-operator-7cfcbd8b56-gdw9x 1/1 Running 0 8m17s 10.244.0.14 kind-control-plane <none> <none> prom-kube-state-metrics-548567f5c6-mn6tl 1/1 Running 0 8m17s 10.244.0.15 kind-control-plane <none> <none> prometheus-prom-kube-prometheus-stack-prometheus-0 2/2 Running 0 8m10s 10.244.0.17 kind-control-plane <none> <none>

apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: rabbitmq # If labels are defined in spec.serviceMonitorSelector.matchLabels of your deployed Prometheus object, make sure to include them here.
metadata.labels to match my Prometheus spec.serviceMonitorSelector.matchLabels config, it began to work again : )master branch.
1 year ago
1 year ago
1 year agoW0621 22:12:13.029860 216556 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
KUBE_PROMETHEUS_STACK_VERSION to the latest i.e. 36.0.2 the quickstart.sh fails for a different reason:
1 year ago
1 year ago
1 year agoRABBITMQ_ENABLED_PLUGINS_FILE in the OCI image the same as for a standard Linux installation (/etc/rabbitmq/enabled_plugins)?docker run -ti --rm --name rabbit rabbitmq:3.10-management
docker exec -ti rabbit bash root@be6c704f8771:/# echo $RABBITMQ_ENABLED_PLUGINS_FILE root@be6c704f8771:/# cat /etc/rabbitmq/enabled_plugins [rabbitmq_management,rabbitmq_prometheus].
apiVersion: v1
kind: Secret
metadata:
name: rabbitmq-validating-webhook-configuration-overlay
namespace: tap-install
annotations:
kapp.k14s.io/change-group: rabbitmq/package-overlays
stringData:
rabbitmq-validating-webhook-configuration-overlay.yaml: |
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.subset({"metadata":{"name":"validating-webhook-configuration"}, "kind": "ValidatingWebhookConfiguration"})
---
metadata:
#@overlay/replace
name: rabbitmq-validating-webhook-configuration
12 months ago
It’s a miss from our team. This is already fixed in the latest OSS release if you are able to install that: https://github.com/rabbitmq/messaging-topology-operator/releases/tag/v1.7.1 (edited)
12 months agoapiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
name: tanzu-rabbitmq
annotations:
ext.packaging.carvel.dev/ytt-paths-from-secret-name.0: rabbitmq-validating-webhook-configuration-overlay # annotation
spec:
serviceAccountName: rabbitmq-operator # Replace with service account name
packageRef:
refName: rabbitmq.tanzu.vmware.com
versionSelection:
constraints: 1.3.0 carvel/kapp over kubectl apply 2. In case there is a client like, say, a bank, paranoid of installing custom software, is there a procedure of how to deploy the RabbitMQ without carvel - ?
12 months ago
Hi @ndmitry, there are some carvel blog posts I could find on their designs and what they aim to solve: https://carvel.dev/blog/kubecon-na21-keynote-blog/
. I’m afraid “Why Carvel” will be explained much better by folks in channel #project-carvel or #tap-assist. (edited)
12 months agokubectl, however the team cannot make any promises on providing support for that. (edited)
12 months ago
(edited) bash script automating the installation of Tanzu RabbitMQ on K8S with all the pre-requisites. I want to know what your thoughts are. It's currently on Github as it exposes no secrets (registry user and password must be externally passed as command line arguments). https://github.com/dmitrynovik/deploy-tanzu-rabbitmq-for-kubernetes
11 months agoecho "INSTALLING HELM..."
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod +x get_helm.sh
./get_helm.sh
4:27 AM
11 months ago
11 months ago
11 months ago
11 months ago
11 months ago
11 months agoOpen ⟶ Waiting on R&D
>Please note that the customer was able to reproduce the issue on multiple foundations.Here is a pointer to the Artifacts.https://drive.google.com/drive/folders/1_rI7TR7U8MhRXYwcF8BPqJL_5xMKsSdo?usp=sharingWaiting on Support ⟶ Waiting on R&D
>Please note that the customer was able to reproduce the issue on multiple foundations.Here is a pointer to the Artifacts.https://drive.google.com/drive/folders/1_rI7TR7U8MhRXYwcF8BPqJL_5xMKsSdo?usp=sharing "user_name": "ci",
"finished_at": "2022-07-11 17:06:03 UTC",
"started_at": "2022-07-11 09:01:40 UTC",
"status": "succeeded",
"additions": [
{
"identifier": "cf",
"label": "VMware Tanzu Application Service",
"product_version": "2.10.32",
"deployment_status": "successful",
"deploy_order": 1,
"change_type": "update",
"guid": "cf-35ab05549bd60b16a581",
"started_at": "2022-07-11 09:05:00 UTC",
"finished_at": "2022-07-11 13:45:39 UTC"
},
stemcells:
+ - alias: bosh-azure-hyperv-ubuntu-xenial-go_agent
+ os: ubuntu-xenial
+ version: '621.245'
- - alias: bosh-azure-hyperv-ubuntu-xenial-go_agent
- os: ubuntu-xenial
- version: '621.236' "user_name": "yuichi_yagi@kamereon.io",
"finished_at": "2022-07-15 14:13:00 UTC",
"started_at": "2022-07-15 10:01:17 UTC",
"status": "succeeded",
"additions": [
{
"identifier": "cf",
"label": "VMware Tanzu Application Service",
"product_version": "2.10.32",
"deployment_status": "successful",
"deploy_order": 1,
"change_type": "update",
"guid": "cf-35ab05549bd60b16a581",
"started_at": "2022-07-15 10:16:30 UTC",
"finished_at": "2022-07-15 12:32:09 UTC"
},2022-07-15 14:38:08.490 [warning] <0.16771.0> Federation exchange 'kemuri' in vhost 'BEDEV' did not connect to exchange 'kemuri' in vhost '13a42cd2-b970-43c1-a5f5-bacd69f297ca' on amqp://q-s0.rabbitmq-server.dynamic-services.service-instance-13a42cd2-b970-43c1-a5f5-bacd69f297ca.bosh/13a42cd2-b970-43c1-a5f5-bacd69f297ca. Reason: {error,econnrefused} 2022-07-15 14:38:08.490 [info] <0.16770.0> supervisor: {<0.16770.0>,rabbit_federation_link_sup}, errorContext: child_terminated, reason: {shutdown,restart}, offender: [{pid,<0.16771.0>},{id,{upstream,[{encrypted,<<"Wj9pwjT/6EPDlD3hGMr24W/pU6NtrwIu7fGXsfPFDr5Hzy5ze9dpd7/ny+Q7i6HadhFMlPEnXm8LiapN2V3G/zhAjqk1CB7liPML9GflFCySUjSwFOoJ1Pr5nirj7Ix+3rzQdz9g575G+YNBHTER49A+emqoYR9dFmYdOObaekk4Zn67oa/JMinh74JzscuZ4Szui5kYqSxL/EJZSgr5YKDZPwWC5W0wF8omAQWZc+ih6tFp9cgqJ2PsLGLPP15i6QPh9pps9cDGTdIfZspKKNwiWyfChATKYtWcQYKGw+jeFK7IlSb5X/DQ6bAuFNXq">>}],<<"kemuri">>,<<"kemuri">>,<<"federation-link-federation-upstream-BEDEV">>,1000,1,5,none,none,false,'on-confirm',none,<<"federation-upstream-BEDEV">>,false,default,multiple}},{mfargs,{rabbit_federation_exchange_link,start_link,[{{upstream,[{encrypted,<<"Wj9pwjT/6EPDlD3hGMr24W/pU6NtrwIu7fGXsfPFDr5Hzy5ze9dpd7/ny+Q7i6HadhFMlPEnXm8LiapN2V3G/zhAjqk1CB7liPML9GflFCySUjSwFOoJ1Pr5nirj7Ix+3rzQdz9g575G+YNBHTER49A+emqoYR9dFmYdOObaekk4Zn67oa/JMinh74JzscuZ4Szui5kYqSxL/EJZSgr5YKDZPwWC5W0wF8omAQWZc+ih6tFp9cgqJ2PsLGLPP15i6QPh9pps9cDGTdIfZspKKNwiWyfChATKYtWcQYKGw+jeFK7IlSb5X/DQ6bAuFNXq">>}],<<"kemuri">>,<<"kemuri">>,<<"federation-link-federation-upstream-BEDEV">>,1000,1,5,none,none,false,'on-confirm',none,<<"federation-upstream-BEDEV">>,false,default,multiple},{resource,<<"BEDEV">>,exchange,<<"kemuri">>}}]}},{restart_type,{permanent,5}},{shutdown,300000},{child_type,worker}] 2022-07-15 14:38:08.491 [error] <0.16770.0> Supervisor {<0.16770.0>,rabbit_federation_link_sup} had child {upstream,[{encrypted,<<"Wj9pwjT/6EPDlD3hGMr24W/pU6NtrwIu7fGXsfPFDr5Hzy5ze9dpd7/ny+Q7i6HadhFMlPEnXm8LiapN2V3G/zhAjqk1CB7liPML9GflFCySUjSwFOoJ1Pr5nirj7Ix+3rzQdz9g575G+YNBHTER49A+emqoYR9dFmYdOObaekk4Zn67oa/JMinh74JzscuZ4Szui5kYqSxL/EJZSgr5YKDZPwWC5W0wF8omAQWZc+ih6tFp9cgqJ2PsLGLPP15i6QPh9pps9cDGTdIfZspKKNwiWyfChATKYtWcQYKGw+jeFK7IlSb5X/DQ6bAuFNXq">>}], <<"federation-link-federation-upstream-BEDEV">>,1000,1,5,none,none, false,'on-confirm',none,<<"federation-upstream-BEDEV">>,false, default,multiple} started with rabbit_federation_exchange_link:start_link({{upstream,[{encrypted,<<"Wj9pwjT/6EPDlD3hGMr24W/pU6NtrwIu7fGXsfPFDr5Hzy5ze9dpd7/ny+Q7i6HadhFMlP...">>}],...},...}) at <0.16771.0> exit with reason {shutdown,restart} in context child_terminated
3.8.30 from 3.8.15 or earlier results in the same issue and the workaround will still have to be applied. I hoped that would provide another workaround but it doesn't.", Is this the only workaround to fix this issue?Do we have a permanent solution on it?VESC-1036Repository with code and instructions to reproduce the issue is located here:https://github.com/lukebakken/vesc-1036#reproducing-vesc-1036TL;DR when two clusters that have a federation link between them are upgraded the federation link does not re-start.warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)Error: unable to perform an operation on node 'rabbit@f433ac483e29f73ebe53a2ec73bc6b26'. Please see diagnostics information and suggestions below. attempted to contact: [rabbit@f433ac483e29f73ebe53a2ec73bc6b26]rabbit@f433ac483e29f73ebe53a2ec73bc6b26: * connected to epmd (port 4369) on f4… warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)Error: unable to perform an operation on node 'rabbit@f433ac483e29f73ebe53a2ec73bc6b26'. Please see diagnostics information and suggestions below. attempted to contact: [rabbit@f433ac483e29f73ebe53a2ec73bc6b26]rabbit@f433ac483e29f73ebe53a2ec73bc6b26: * connected to epmd (port 4369) on f433ac483e29f73ebe53a2ec73bc6b26 * epmd reports: node 'rabbit' not running at all no other nodes on f433ac483e29f73ebe53a2ec73bc6b26 * suggestion: start the nodewarning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)2022-07-22 15:46:56.443245+00:00 [warning] <0.229.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,2022-07-22 15:46:56.443245+00:00 [warning] <0.229.0> [rabbit@56622490f61c9428f6513720349de4cb,2022-07-22 15:46:56.443245+00:00 [warning] <0.229.0> rabbit@f433ac483e29f73ebe53a2ec73bc6b26,2022-07-22 15:46:56.443245+00:00 [warning] <0.229.0> rabbit@6e83f75aa40fd2b31415bbd66ac6c83b], It seems like a node needs to be started somewhere? But, I’m not seeing this node referenced in the errors as a vm in the rabbit mq deployment. Does anyone know what these errors mean and how to resolve?
11 months ago
11 months agobackground_gc_enabled = true in rabbitmq.conf should help. The rates on the screenshot are low, so my guess is that it is a similar workload. (edited)
11 months ago## Whether or not to enable background periodic forced GC runs for all
## Erlang processes on the node in "waiting" state.
##
## Disabling background GC may reduce latency for client operations,
## keeping it enabled may reduce median RAM usage by the binary heap
11 months ago
11 months ago
11 months ago
11 months agoI got a ticket from a app team asking if we could do the following: It is found that MQ instance is exposed on the public network and can be accessed through this address:https://rmq-380e34ea-3950-4439-94d7-50636c3c9e10.sys.pd01.cneast.cf.ford.com.cn/.There may be critical security risks. although i konw pcf MQ is PAAS ,Is there any way to disable the management portal like the "rabbitmq-plugins disable rabbitmq_management "operation in open source version Now I believe we can disable it by un-checking the box in the tile, but that would be a global change (so it would affect everyone using rabbit mq). So I am wondering, do we need the management endpoint for rabbit mq? I think we do, but wanted verification from VMware.
registry.tanzu.vmware.com our tests fail because the rabbitmqc-server’s init container fails with the following error:failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "sh": executable file not found in $PATH: unknown
. Does anyone ever seen an issue like this? Thanksimgpkg copy and I see the sha for tanzu-rabbitmq on GCR is 8dafa8c4b618cbb90d7ee94f14401e7c09da29da8842dc12c64e6b1aa8ebd4e8 and I pulled the image with impkg pull and I also see the same sha:➜ imgpkg pull -b registry.tanzu.vmware.com/p-rabbitmq-for-kubernetes/tanzu-rabbitmq-package-repo:1.3.0 -o /tmp/tanzurabbitmq-img Pulling bundle 'registry.tanzu.vmware.com/p-rabbitmq-for-kubernetes/tanzu-rabbitmq-package-repo@sha256:8dafa8c4b618cbb90d7ee94f14401e7c09da29da8842dc12c64e6b1aa8ebd4e8' Extracting layer 'sha256:1c41e2d8fddef1920f6c2e210a086d6d327d7681650e1ea32cce2d7e9d097929' (1/1)
9 months agorabbitmq_sharing does not replicate, so yes, it uses classic queues and it makes limited sense to make QQs an option there. Replications defeats the purpose of sharding to a large degree.RabbitMQ is asked to stop":21bcd8fc-7013-482c-b1f2-9be782056610==> crash.log.1 <==
2022-09-17 03:14:48 =SUPERVISOR REPORT====
Supervisor: {<0.16765.0>,rabbit_channel_sup_sup}
Context: shutdown_error
Reason: shutdown
Offender: [{nb_children,1},{id,channel_sup},{mfargs,{rabbit_channel_sup,start_link,[]}},{restart_type,temporary},{shutdown,infinity},{child_type,supervisor}]
2022-09-17 03:14:48 =SUPERVISOR REPORT====
Supervisor: {<0.16663.0>,rabbit_channel_sup_sup}
Context: shutdown_error
Reason: shutdown
2022-09-17 03:13:58.276 [info] <0.17048.0> connection <0.17048.0> (10.70.2.74:44358 -> 10.70.6.19:5672 - SpringAMQP#16e9206e:1): user '47228463-72de-4cdd-ada8-e942e1fa1037' authenticated and granted access to vhost 'ac8666aa-d528-438e-981e-ca7cee867b0b'
2022-09-17 03:14:03.937 [info] <0.528.0> node rabbit@ba0fd340da9f5a56343e16c134539ef9 up
2022-09-17 03:14:10.851 [info] <0.528.0> rabbit on node rabbit@ba0fd340da9f5a56343e16c134539ef9 upRabbitMQ is asked to stop...2022-09-17 03:14:47.968 [info] <0.17297.0> Stopping RabbitMQ applications and their dependencies in the following order:
89072287-19e1-4743-9227-279047d53e8d==> crash.log.1 <==
2022-09-17 03:16:20 =SUPERVISOR REPORT====
Supervisor: {<0.10901.0>,rabbit_channel_sup_sup}
Context: shutdown_error
Reason: shutdown
Offender: [{nb_children,3},{id,channel_sup},{mfargs,{rabbit_channel_sup,start_link,[]}},{restart_type,temporary},{shutdown,infinity},{child_type,supervisor}]
2022-09-17 03:16:20 =SUPERVISOR REPORT====
Supervisor: {<0.12916.0>,rabbit_channel_sup_sup}
Context: shutdown_error
Reason: shutdown
2022-09-17 03:15:48.048 [error] <0.13650.0> Channel error on connection <0.13627.0> (10.70.2.99:51438 -> 10.70.6.20:5672, vhost: 'ac8666aa-d528-438e-981e-ca7cee867b0b', user: '93457512-55bb-4ef9-a74e-887533a5c087'), channel 1:
operation queue.declare caused a channel exception not_found: home node 'rabbit@f404632403d8df22de052507a799e721' of durable queue 'CMCP.ACCESSLOG' in vhost 'ac8666aa-d528-438e-981e-ca7cee867b0b' is down or inaccessible
2022-09-17 03:15:49.080 [info] <0.525.0> rabbit on node rabbit@f404632403d8df22de052507a799e721 up2022-09-17 03:16:20.175 [info] <0.13835.0> RabbitMQ is asked to stop...2022-09-17 03:16:20.514 [info] <0.13835.0> Stopping RabbitMQ applications and their dependencies in the following order:
fae8339d-f211-499d-af1e-b9ddfc03416a
==> crash.log.1 <==
2022-09-17 03:13:10 =SUPERVISOR REPORT====
Supervisor: {<0.15265.0>,rabbit_channel_sup_sup}
Context: shutdown_error
Reason: shutdown
Offender: [{nb_children,1},{id,channel_sup},{mfargs,{rabbit_channel_sup,start_link,[]}},{restart_type,temporary},{shutdown,infinity},{child_type,supervisor}]
2022-09-17 03:13:10 =SUPERVISOR REPORT====
Supervisor: {<0.9099.0>,rabbit_channel_sup_sup}
Context: shutdown_error
Reason: shutdown
2022-09-17 03:02:04.240 [info] <0.15261.0> Connection <0.15261.0> (10.70.2.106:58574 -> 10.70.6.18:5672) has a client-provided name: SpringAMQP#373bfbf6:0
2022-09-17 03:02:04.247 [info] <0.15261.0> connection <0.15261.0> (10.70.2.106:58574 -> 10.70.6.18:5672 - SpringAMQP#373bfbf6:0): user '71d3f224-e455-42ea-8cd2-eab90235ce2b' authenticated and granted access to vhost 'ac8666aa-d528-438e-981e-ca7cee867b0b'RabbitMQ is asked to stop...2022-09-17 03:13:10.602 [info] <0.15552.0> Stopping RabbitMQ applications and their dependencies in the following order:
2022-06-02 21:02:15.045 [info] <0.6052.2355> RabbitMQ is asked to stop...
RabbitmqCluster object may have some clues, or the events in the Kubernetes nodes, or the kubelet, or the kube-api server.I’m a bit surprised to see RabbitMQ 3.8.7, given that CapGroup is in Tanzu RabbitMQ for k8s 1.2.2, according to your original message. Specially because the minimum version that works with the Cluster Operator is 3.8.8. It also looks like they are not using Tanzu RabbitMQ, but OSS Ra… Error: this command requires the target node to be stopped,rabbitmqctl -n rabbit@node-0 forget_cluster_node --offline rabbit@node-1
RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the --erlang-cookie switch instead.
Error: this command requires the target node to be stopped.
Arguments given:
-n rabbit@rabbitmq-sura-rabbitmq-ha-1.rabbitmq-sura-rabbitmq-ha-discovery.rabbitmq.svc.srncsk8mspdn.suramericana.com.co forget_cluster_node --offline rabbit@rabbitmq-sura-rabbitmq-ha-0.rabbitmq-sura-rabbitmq-ha-discovery.rabbitmq.svc.srncsk8mspdn.suramericana.com.co
Usage
rabbitmqctl [--node <node>] [--longnames] [--quiet] forget_cluster_node [--offline] <existing_cluster_member_node>rabbitmqctl status it says that this command requires the 'rabbit' app to be running on the target node.
9 months ago







)
9 months ago
9 months agokapp deploy (packagerepository) fails with authorization problem:kubectl get packagerepositories/tanzu-rabbitmq-repo -n rabbitmq-system -o=jsonpath={.status.usefulErrorMessage}
vendir: Error: Syncing directory '0':
Syncing directory '.' with imgpkgBundle contents:
Imgpkg: exit status 1 (stderr: imgpkg: Error: Fetching image:
GET https://registry.tanzu.vmware.com/v2/p-rabbitmq-for-kubernetes/tanzu-rabbitmq-package-repo/manifests/1.3.0:
UNAUTHORIZED: unauthorized to access repository: p-rabbitmq-for-kubernetes/tanzu-rabbitmq-package-repo, action: pull: unauthorized to access repository: p-rabbitmq-for-kubernetes/tanzu-rabbitmq-package-repo, action: pull
)carvel/imgpkg in order to diagnose why above happens? (edited) <resource-name>-fetch-<i>apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
name: tanzu-rabbitmq-repo
spec:
fetch:
imgpkgBundle:
image: registry.tanzu.vmware.com/p-rabbitmq-for-kubernetes/tanzu-rabbitmq-package-repo:1.3.2
secretRef:
name: reg-credsIMGPKG_ENABLE_IAAS_AUTH environment variable to false on the kapp-controller Podkubectl get secret -n rabbitmq-system NAME TYPE DATA AGE default-token-dq99t kubernetes.io/service-account-token 3 13d rabbitmq-token-xpcmp kubernetes.io/service-account-token 3 13d reg-creds kubernetes.io/dockerconfigjson 1 13d tanzu-rabbitmq-repo-fetch-0 kubernetes.io/dockerconfigjson 1 13d
reg-creds that I create explicitly has correct credentials encoded as JWT whereas the generated tanzu-rabbitmq-repo-fetch-0 has empty auth ( .dockerconfigjson: eyJhdXRocyI6e319 ), obviously this is a problem but not sure why this is happening... any idea? Note again, this is only happening in my AKS test, when running on kind locally the tanzu-rabbitmq-repo-fetch-0 has correct creds (edited)
8 months ago
8 months agoOpen ⟶ Waiting on R&Drabbitmqctl cluster_status | grep "Cluster name"curl -s localhost:15692/metrics | grep identityrabbitmqctl -q set_cluster_name <Cluster Name> [root@rmqsrv0 ~]# curl -s localhost:15692/metrics | grep identity
# TYPE rabbitmq_identity_info untyped
# HELP rabbitmq_identity_info RabbitMQ node & cluster identity info
rabbitmq_identity_info{rabbitmq_node="rabbit@rmqsrv0",rabbitmq_cluster="rabbitmq-v3.11.2-cluster",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-8dzV2ZXuAItZz_p-AxlAsQ"} 1
[root@rmqsrv1 ~]# curl -s localhost:15692/metrics | grep identity
# TYPE rabbitmq_identity_info untyped
# HELP rabbitmq_identity_info RabbitMQ node & cluster identity info
rabbitmq_identity_info{rabbitmq_node="rabbit@rmqsrv1",rabbitmq_cluster="rabbitmq-v3.11.2-cluster",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-8dzV2ZXuAItZz_p-AxlAsQ"} 1
[root@rmqsrv2 ~]# curl -s localhost:15692/metrics | grep identity
# TYPE rabbitmq_identity_info untyped
# HELP rabbitmq_identity_info RabbitMQ node & cluster identity info
rabbitmq_identity_info{rabbitmq_node="rabbit@rmqsrv2",rabbitmq_cluster="rabbitmq-v3.11.2-cluster",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-8dzV2ZXuAItZz_p-AxlAsQ"} 1Waiting on Support ⟶ Waiting on R&D[upgrade-all-service-instances] 2022/11/01 08:03:45.388296 [upgrade-all] [99c12aa6-3aa9-480c-98d2-1ff985b0da9d] Result: instance already up to date - operation skipped[Environment]RabbitMQ: 3.9.20
Support Bundle[Slack Thread]https://vmware.slack.com/archives/C0RDGG81Z/p1667515510565539
rmq manifest.yml
vm-extension
Service instance logs
Pictures showing the change and the result
New Service instance working artifacts
ICYMI @Arnab Chakraborty published recently https://tanzu.vmware.com/content/blog/vmware-improves-on-open-source-rabbitmqWaiting on Support ⟶ Waiting on R&D2022-11-18 00:45:05 /Users/kli/pre-tool/bin/update-service-tiles.sh:254 DEBUG ### stage-product <p-rabbitmq-2.1.0-build.13.pivotal> failed (1), retrying ...Customer tried multiple versions of the RMQ tile as the target version (2.1.0 - 2.1.5) and all failed with the same error. This issue is blocking the customer’s upgrade and support has expired for the version that they are running in production. It is urgent that they get this issuse resolved.Opsman 2.10.47TAS 2.11.22RMQ 2.0.16
staging p-rabbitmq 2.1.0-build.13
2022/11/18 00:45:27 failed to stage product: request failed: unexpected response from /api/v0/staged/products/p-rabbitmq-90fe13570110d20a852f:
HTTP/1.1 422 Unprocessable Entity
Transfer-Encoding: chunked
Cache-Control: private, no-store
Connection: keep-alive
Content-Security-Policy: script-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; object-src 'none';
Content-Type: application/json; charset=utf-8
Date: Thu, 17 Nov 2022 16:45:09 GMT
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Referrer-Policy: strict-origin-when-cross-origin
Server: Ops Manager
Strict-Transport-Security: max-age=63072000; includeSubDomains
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: a95eed77-b497-436a-a0a7-ddf88633f22f
X-Runtime: 0.961145
d2
{"errors":["Could not apply migrations. '201701301408_hapolicy.js' has javascript errors 'TypeError [Error]: Cannot read properties of undefined (reading 'value')'. Please contact your VMware representative."]}
7 months ago
7 months ago
7 months ago
7 months ago
7 months ago
7 months ago
7 months agorabbitmq-queues rebalance "all" and redistributing the Leader queues, does quorum have a significant advantage over mirror queues in selecting new Leader nodes and activating new Leader nodes? Will quorum queues resume service in less time than mirror queues?
7 months ago
7 months agohttps://[hostname]:15672/api/health/checks/port-listener/5671and needs some help. I don't see any doc for this particular scenario. can someone point me in the right direction? Thank you!-1 .time.Tick() by time.NewTicker() to process scrape_interval. The later would panic on non-positive value.
Open ⟶ Waiting on R&D-1 .time.Tick() by time.NewTicker() to process scrape_interval. The later would panic on non-positive value.
[upgrade-all-service-instances] 2023/01/09 16:38:40.965250 no valid U token found in cache, obtaining a new one [upgrade-all-service-instances] 2023/01/09 16:38:41.004010 obtained U token, expires in 3599 seconds upgrade-all-service-instances2023/01/09 16:38:41.004271 GET https://api.sys.<redacted>/v2/info Luperade-all-service-instances] 2023/01/09 16:38:41.030539 Muograde-alll Uperading all instances via CF upgrade-all-service-instances2023/01/09 16:38:41.030703 upgrade-all]STARTING OPERATION with 3 concurrent workers [upgrade-all-service-instances] 2023/01/09 16:38:41.194102 [upgrade-all] Service Instances: ad437b-603-459b-6446-dcb4925905f3 5f755c22-e00c-44eb-9aae-6e704e0e3271 7c62be40-5bda-4131-90de-6164f86283fa 41ab6 565-035f-4895-a036-3d218839143 8842dcc2-8eee-4adc-8d2-945552296048 13081625-5510-43da-bd30-4d4f19cd9df9 590c09ba-4585-4f3f -aa64-f9777822698 dfcaf221-d615-4a67-99ae-2743359a1cca 842aacad-4c58-4ab9-97bb-cOa3ce80e009 a 30c90db-1a7c-4ffd-887d-edb00df10a12 7cf52e33-483e-4032-af54-0d97bfc9594b 63049613-ff82-4ef9-a827-c4d0e6b52791 f06f1bce-cbob-416b-b3ce-ff615ecc7eld ae2a2c16-a356-4773-815f-d1c937dc5c5b 396c0f8e-085f-4ff9-bbf0-852a92ac8e 8f [upgrade-all-service-instances 2023/01/09 16:38:41.194116 upgrade-alll Total Service Instances found: 15 Lupgrade-all-service-instancesl2023/01/09 16:38:41.194121 [upgrade-alllSTARTINGCANARIES:1 canaries [upgrade-all-service-instances 2023/01/09 16:38:41.194125 upgrade-all Processing all canaries. Attempt 1/5 [upgrade-all-service-instances] 2023/01/09 16:38:41.194136 [upgrade-all] [ad437b-603-459b-b446-dcb4925905f3] Starting to process service instance [upgrade-all-service-instances] 2023/01/09 16:38:41.358995 GET https://api.sys.<redacted>/v2/service_plans?q service_instance_guid:ad437bf-603-459b-b446-dcb4925905f3 [upgrade-all-service-instances] 2023/01/09 16:38:41.398356 PUT https://api.sys.<redacted>/v2/service_instances/ad437b-603-459b-b446-dcb4925905f3?accepts_incomplete=true [upgrade-all-service-instances] 2023/01/09 16:38:41.422100 [upgrade-all] FINISHED PROCESSING Status: FAILED; Summary: Number of successful operations: 0; Number of skipped operations: 0; Number of service ins tance orphans detected: 0; Number of deleted instances before operation could happen: 0; Number of busy instances which could not be processed: 0; Number of service instances that failed to process: 1 [ad437b-603-45 96-b446-dcb4925905f3] Tunerade-all-service-instancesl2023/01/09 16:38:41.422113 canaries didn't process successfullv: failed to trigger oneration for instance "ad43e7hf-6e03-459b-h446-dch4925905f3": unexnected resnonse status 422 when upgrading service instance "ad43e7bf-603-459b-b446-dcb4925905f3; response body "{In \"description\": |"The service broker does not support upgrades for service instances created from this plan. \", In \"error_co del": \"CF-MaintenanceInfoNotSupported\", In "code\": 390006\n| \n"
I just got a bunch of emails, that someone has used a github recovery code, changed the account's (cf-london?) password and added a ssh key. Has anyone of y'all done that? Or is this the
breach hitting us?
5 months agoGolang to access RabbitMQ , and it report an error which is x509: certificate is not valid for any names, but wanted to match localhost .I don't know what happened. Is my certificate generated wrong? I didn't find the missing config item. please help me & thanks a lot.
5 months ago
5 months ago[ client_alt_names ]
DNS.1 = $common_name
DNS.2 = $client_alt_name
DNS.3 = localhost
# examples of more Subject Alternative Names
5 months ago
5 months agorabbitmq-server/9ff8188c-0ab4-4187-b8f8-e0213e7dbfb0:/var/vcap/jobs/loggr-syslog-agent/config/certs# ls -lResults in errors like
total 40
-rw-r----- 1 root vcap 1213 Jan 25 16:20 cache_ca.crt
-rw-r----- 1 root vcap 1254 Jan 25 16:20 cache_client.crt
-rw-r----- 1 root vcap 1676 Jan 25 16:20 cache_client.key
-rw-r----- 1 root vcap1 Jan 25 16:20 drain_ca.crt
-rw-r----- 1 root vcap 1213 Jan 25 16:20 loggregator_ca.crt
-rw-r----- 1 root vcap 1197 Jan 25 16:20 metrics_ca.crt
-rw-r----- 1 root vcap 1242 Jan 25 16:20 metrics.crt
-rw-r----- 1 root vcap 1676 Jan 25 16:20 metrics.key
-rw-r----- 1 root vcap 1246 Jan 25 16:20 syslog_agent.crt
-rw-r----- 1 root vcap 1676 Jan 25 16:20 syslog_agent.key
2023/01/18 14:47:58 failed to write to log-cache.service.cf.internal:6067, retrying in 2.048s, err: x509: certificate signed by unknown authorityThis has been observed before in isolation segments:
Open ⟶ Waiting on R&Drabbitmq-server/9ff8188c-0ab4-4187-b8f8-e0213e7dbfb0:/var/vcap/jobs/loggr-syslog-agent/config/certs# ls -lResults in errors like
total 40
-rw-r----- 1 root vcap 1213 Jan 25 16:20 cache_ca.crt
-rw-r----- 1 root vcap 1254 Jan 25 16:20 cache_client.crt
-rw-r----- 1 root vcap 1676 Jan 25 16:20 cache_client.key
-rw-r----- 1 root vcap1 Jan 25 16:20 drain_ca.crt
-rw-r----- 1 root vcap 1213 Jan 25 16:20 loggregator_ca.crt
-rw-r----- 1 root vcap 1197 Jan 25 16:20 metrics_ca.crt
-rw-r----- 1 root vcap 1242 Jan 25 16:20 metrics.crt
-rw-r----- 1 root vcap 1676 Jan 25 16:20 metrics.key
-rw-r----- 1 root vcap 1246 Jan 25 16:20 syslog_agent.crt
-rw-r----- 1 root vcap 1676 Jan 25 16:20 syslog_agent.key
2023/01/18 14:47:58 failed to write to log-cache.service.cf.internal:6067, retrying in 2.048s, err: x509: certificate signed by unknown authorityThis has been observed before in isolation segments:
Done ⟶ Waiting on R&D-1 .time.Tick() by time.NewTicker() to process scrape_interval. The later would panic on non-positive value.
4 months ago2023-02-20 15:31:22.552269+00:00 [info] <0.11950.1> queue 'escid.esp.commonlogger.contracts.messages.shopmessage:esp-commonlogger' in vhost '/': Leader monitor down with {system_limit,[{erlang,spawn,[erlang|...],[...]},{erlang,spawn,1,...},{rabbit_quorum_queue,handle_tick,...},{ra_server_proc,...},{...}|...]}, setting election timeout
2023-02-20 15:31:13.393219+00:00 [error] <13943.31034.48> Too many processes
2023-02-20 15:31:13.393219+00:00 [error] <13943.31034.48>
4 months ago
4 months ago
4 months ago---
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
...
spec:
replicas: 1
rabbitmq:
envConfig: |
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+P 10000000"2023-02-28 23:40:40.554000+00:00 [warning] <0.10276.0> *** Publishers will be blocked until this alarm clears *** 2023-02-28 23:40:49.565043+00:00 [warning] <0.10276.0> memory resource limit alarm cleared on node 'rabbit@ab1118af-78da-46a0-813f-32e383063915.rabbitmq-server.services.service-instance-85b1f478-9427-41bc-8a73-28b1eb9bfb27.bosh' 2023-02-28 23:40:49.565190+00:00 [warning] <0.10276.0> memory resource limit alarm cleared across the cluster 2023-02-28 23:40:53.568861+00:00 [warning] <0.10276.0> memory resource limit alarm set on node 'rabbit@ab1118af-78da-46a0-813f-32e383063915.rabbitmq-server.services.service-instance-85b1f478-9427-41bc-8a73-28b1eb9bfb27.bosh'. 2023-02-28 23:40:53.568861+00:00 [warning] <0.10276.0> *** Publishers will be blocked until this alarm clears *** 2023-02-28 23:41:00.576090+00:00 [warning] <0.10276.0> memory resource limit alarm cleared on node 'rabbit@ab1118af-78da-46a0-813f-32e383063915.rabbitmq-server.services.service-instance-85b1f478-9427-41bc-8a73-28b1eb9bfb27.bosh' 2023-02-28 23:41:00.576249+00:00 [warning] <0.10276.0> memory resource limit alarm cleared across the cluster 2023-02-28 23:41:09.585286+00:00 [warning] <0.10276.0> memory resource limit alarm set on node 'rabbit@ab1118af-78da-46a0-813f-32e383063915.rabbitmq-server.services.service-instance-85b1f478-9427-41bc-8a73-28b1eb9bfb27.bosh'. 2023-02-28 23:41:09.585286+00:00 [warning] <0.10276.0> *** Publishers will be blocked until this alarm clears *** 2023-02-28 23:41:28.612786+00:00 [warning] <0.10276.0> memory resource limit alarm cleared on node 'rabbit@ab1118af-78da-46a0-813f-32e383063915.rabbitmq-server.services.service-instance-85b1f478-9427-41bc-8a73-28b1eb9bfb27.bosh' 2023-02-28 23:41:28.612925+00:00 [warning] <0.10276.0> memory resource limit alarm cleared across the cluster
Total memory used: 4.9322 gb Calculation strategy: rss Memory high watermark setting: 0.5 of available memory, computed to: 16.6221 gb
=erl_crash_dump:0.5 Tue Feb 28 23:17:35 2023 Slogan: eheap_alloc: Cannot allocate 16925484064 bytes of memory (of type "heap"). System version: Erlang/OTP 24 [erts-12.3.2.6] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] Taints: asn1rt_nif,crypto Atoms: 49480 Calling Thread: scheduler:0 =scheduler:1 Scheduler Sleep Info Flags: SLEEPING | TSE_SLEEPING | WAITING Scheduler Sleep Info Aux Work:
total used free shared buff/cache available Mem: 30G 2.9G 14G 13M 13G 27G Swap: 1.0G 3.5M 1.0G
-1 failing. loggregator-agent-release v6.5.6 fixes this. 2023/03/16 11:09:59 starting Prom Scraper... FIELD NAME: TYPE: ENV: REQUIRED: VALUE: Config.UseRFC3339 bool USE_RFC3339 false (OMITTED) Config.ClientKeyPath string CLIENT_KEY_PATH true /var/vcap/jobs/prom_scraper/config/certs/loggregator_agent.key Config.ClientCertPath string CLIENT_CERT_PATH true /var/vcap/jobs/prom_scraper/config/certs/loggregator_agent.crt Config.CACertPath string CA_CERT_PATH true /var/vcap/jobs/prom_scraper/config/certs/loggregator_ca.crt Config.ScrapeKeyPath string SCRAPE_KEY_PATH false /var/vcap/jobs/prom_scraper/config/certs/scrape.key Config.ScrapeCertPath string SCRAPE_CERT_PATH false /var/vcap/jobs/prom_scraper/config/certs/scrape.crt Config.ScrapeCACertPath string SCRAPE_CA_CERT_PATH false /var/vcap/jobs/prom_scraper/config/certs/scrape_ca.crt Config.LoggregatorIngressAddr string LOGGREGATOR_AGENT_ADDR true localhost:3458 Config.DefaultSourceID string DEFAULT_SOURCE_ID true rabbitmq-server Config.ConfigGlobs []string CONFIG_GLOBS false [/var/vcap/jobs/*/config/prom_scraper*config.yml /var/vcap/jobs/*/config/metric_port.yml] Config.DefaultScrapeInterval time.Duration SCRAPE_INTERVAL false -1s Config.SkipSSLValidation bool SKIP_SSL_VALIDATION false false MetricsServer.DebugMetrics bool DEBUG_METRICS false false MetricsServer.Port uint16 METRICS_PORT false 14821 MetricsServer.PprofPort uint16 PPROF_PORT false 0 MetricsServer.CAFile string METRICS_CA_FILE_PATH true /var/vcap/jobs/prom_scraper/config/certs/metrics_ca.crt MetricsServer.CertFile string METRICS_CERT_FILE_PATH true /var/vcap/jobs/prom_scraper/config/certs/metrics.crt MetricsServer.KeyFile string METRICS_KEY_FILE_PATH true /var/vcap/jobs/prom_scraper/config/certs/metrics.key 2023/03/16 11:09:59 Metrics endpoint is listening on 127.0.0.1:14821 2023/03/16 11:09:59 closing Prom Scraper...
172.18.255.202 as follows:kubectl -n rabbitmq-system get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE standby-replication-webhook-service ClusterIP 10.96.99.171 <none> 443/TCP 112d tanzu-rabbitmq LoadBalancer 10.96.38.130 172.18.255.202 5672:30095/TCP,15672:32486/TCP,15692:32030/TCP,5552:30679/TCP 112d tanzu-rabbitmq-nodes ClusterIP None <none> 4369/TCP,25672/TCP 112d webhook-service ClusterIP 10.96.126.105 <none> 443/TCP 112d
telnet 172.18.255.202 5672 Trying 172.18.255.202... Connected to 172.18.255.202. Escape character is '^]'.
perf-test, the stream-perf-test, or in fact even my own Spring Boot Java app, it does not connect. For example:docker run -it --rm pivotalrabbitmq/perf-test:latest --uri amqp://admin:admin@172.18.255.202:5672 Main thread caught exception: java.net.SocketTimeoutException: Connect timed out 23:29:54.388 [main] ERROR com.rabbitmq.perf.PerfTest - Main thread caught exception java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:639) at com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:59) at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:63) at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:160) at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1216) at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1173) at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1071) at com.rabbitmq.perf.MulticastSet$ConnectionCreator.createConfigurationConnections(MulticastSet.java:889) at com.rabbitmq.perf.MulticastSet.createConfigurationConnections(MulticastSet.java:354) at com.rabbitmq.perf.MulticastSet.run(MulticastSet.java:205) at com.rabbitmq.perf.PerfTest.main(PerfTest.java:480) at com.rabbitmq.perf.PerfTest.main(PerfTest.java:604) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) test stopped
rabbitmq list_queues?standby.replication.retention.size_limit.messages= 5000000000rabbitmq.internal.osr.metrics and rabbitmq.internal.osr.messages will each take up a maximum of 5GB of disk space, so they will take up a total of 10GB of disk space, right?
3 months agostandby.replication.retention.size_limit.messages, as the name suggests, will only have an effect on rabbitmq.internal.osr.messages (there's one in every standby-enabled virtual host)rabbitmq.internal.osr.metrics queue, it has its own default size_limit or time_limit, right? It doesn't take up unlimited disk space either, right?
3 months agorabbitmq.internal.* streams or and whatnot.iops is not an accepted key by the Ops Manager API. You cannot pass arbitrary cloud properties via that API endpoint. We would need to implement features in order to support that field. An additional field is needed in the disk_types method - PUT /api/v0/disk_types.
3 months ago
3 months agoListing policies for vhost "HP-CI-BCT" ...
vhost name pattern apply-to definition priority
HP-CI-BCT HA2 .* all {"ha-mode":"exactly","ha-params":2,"ha-promote-on-failure":"when-synced","ha-promote-on-shutdown":"when-synced","ha-sync-mode":"automatic","queue-master-locator":"min-masters"} 0
...
Listing queues for vhost Release-Lab ...
name durable auto_delete arguments policy pid owner_pid exclusive exclusive_consumer_pid exclusive_consumer_tag messages_ready messages_unacknowledged messages messages_ready_ram messages_unacknowledged_ram messages_ram messages_persistent message_bytes message_bytes_ready message_bytes_unacknowledged message_bytes_ram message_bytes_persistent head_message_timestamp disk_reads disk_writes consumers consumer_utilisation consumer_capacity memory slave_pids synchronised_slave_pids state type leader members online
federation: Upmc.Timelines -> rabbit@swhthpap5250.acct.upmchs.net:DR-Test-Release-Lab:Upmc.Timelines true false [{"x-internal-purpose","federation"},{"x-queue-type","classic"}] HA2 <rabbit@swhthpap5083.acct.upmchs.net.1679039575.150.5507> false 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 55520 [<rabbit@swhthpap5084.acct.upmchs.net.1679041217.31236.4658>] [<rabbit@swhthpap5084.acct.upmchs.net.1679041217.31236.4658>] running classic
3 months ago
3 months ago
3 months agorabbitmqctl set_parameter federation-upstream -p / fed-up-client-201 '{"uri": "amqp://guest:guest@192.168.67.201:5672", "ack-mode": "on confirm"}'192.168.67.201" in the uri be replaced with a floating IP (generated by HAProxy), or can port 57672 be replaced with the Nginx port (the Nginix port is mapped to port 5672 on all three nodes of the cluster).
2 months ago1. Have we done any tests using Velero to restore RabbitMQ Pods(including queues and messages)?
2. If not, is it theoretically possible to recover RabbitMQ Cluster (including queues and messages) using Veleroh?
Are there any potential risks?
2 months ago
2 months ago
2 months ago
2 months ago
2 months ago
2 months agoroot@idmvrabint1:/# rabbitmqctl report
Reporting server status of node tcmb-test-rabbit@idmvrabint1 ...
Status of node tcmb-test-rabbit@idmvrabint1 ...
Runtime
2 months ago
2 months ago
2 months agoTolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints: topology.kubernetes.io/zone:ScheduleAnyway when max skew 1 is exceeded for selector app.kubernetes.io/name=rabbitmq
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 4m35s default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the conditionapiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rabbitmq
namespace: default
spec:
replicas: 1
service:
type: LoadBalancer
resources:
requests:
cpu: "0.5"
memory: "1Gi"
limits:
cpu: "0.5"
memory: "1Gi"
imagePullSecrets:
- name: tanzu-rabbitmq-registry-creds
rabbitmq:
additionalPlugins:
- rabbitmq_stream_management
- rabbitmq_shovel_management
- rabbitmq_federation_management
- rabbitmq_mqtteksctl utils associate-iam-oidc-provider --region us-east-2 --cluster=vmware-data-cluster --approve eksctl create iamserviceaccount \ --name ebs-csi-controller-sa \ --namespace kube-system \ --cluster vmware-data-cluster \ --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \ --approve \ --role-only \ --role-name AmazonEKS_EBS_CSI_DriverRole eksctl create addon --name aws-ebs-csi-driver --cluster vmware-data-cluster --service-account-role-arn arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):role/AmazonEKS_EBS_CSI_DriverRole --force
%APPDATA%\RabbitMQ\ , however that points to my user profile and the RMQ is installed as a windows service. Where would be the default location of config on a RMQ-as-a-windows-service?UPDATE: I can see the rabbitmq report does tell me that the config is placed under my user profile aka C:\Users\NovikD... . I reckon this is less than ideal because if my user is deactivated, all configuration would be lost. Shouldn't we create the config file not in the current user profile when installing on Windows but somewhere in a more permanent place e.g. in the RabbitMQ installation folder? This applies not just to config default location but also to enabled plugins, mnesia, etc. Thanks. (edited)
2 months ago
2 months ago rabbitmq-queues rebalance quorum command is executed, the rebalance cannot be executed properly.
bitnami@rbt-c2-srv1 [ ~ ]$ rabbitmq-queues rebalance quorum
Re-balancing leaders of quorum queues...
22:50:01.510 [warning] Error migrating queue {:resource, "/", :queue, "test5.quorum"}: :ra_members_timeout
22:50:06.511 [warning] Error migrating queue {:resource, "/", :queue, "test6.quorum"}: :ra_members_timeout
22:50:11.512 [warning] Error migrating queue {:resource, "/", :queue, "test3.quorum"}: :ra_members_timeout
22:50:16.513 [warning] Error migrating queue {:resource, "/", :queue, "test1.quorum"}: :ra_members_timeout
22:50:21.514 [warning] Error migrating queue {:resource, "/", :queue, "test4.quorum"}: :ra_members_timeout
22:50:21.514 [warning] Node :"rabbit@rbt-c2-srv1" contains 5 queues, but all have already migrated. Do nothing
+--------------------+-------------------------+-------------------------------------+
| Node name | Number of quorum queues | Number of replicated classic queues |
+--------------------+-------------------------+-------------------------------------+
| rabbit@rbt-c2-srv2 | 1 | 0 |
+--------------------+-------------------------+-------------------------------------+
| rabbit@rbt-c2-srv1 | 5 | 0 |
+--------------------+-------------------------+-------------------------------------+rabbitmq-queues rebalance quorum command normally when all three nodes are present.bitnami@rbt-c2-srv1 [ ~ ]$ rabbitmq-queues rebalance quorum Re-balancing leaders of quorum queues... +--------------------+-------------------------+-------------------------------------+ | Node name | Number of quorum queues | Number of replicated classic queues | +--------------------+-------------------------+-------------------------------------+ | rabbit@rbt-c2-srv3 | 2 | 0 | +--------------------+-------------------------+-------------------------------------+ | rabbit@rbt-c2-srv2 | 2 | 0 | +--------------------+-------------------------+-------------------------------------+ | rabbit@rbt-c2-srv1 | 2 | 0 | +--------------------+-------------------------+-------------------------------------+
2 months ago
2 months ago"http://rmq-dev-demo-prometheus-metrics-bb00000015-rabbitmq-demo-inf-01.apps.dev-int-sjx1.capgroup.com" , I am getting below error Application is not availableThe application is currently not serving requests at this endpoint. It may not have been started or is still starting.Possible reasons you are seeing this page:The host doesn't exist. Make sure the hostname was typed correctly and that a route matching this hostname exists.The host exists, but doesn't have a matching path. Check if the URL path was typed correctly and that the route was created using the desired path.Route and path matches, but all pods are down. Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.root@RMQ-Ubuntu-VM:~# kubectl -n rabbitmq-system get pods NAME READY STATUS RESTARTS AGE messaging-topology-operator-84bc74db9d-7wc2q 1/1 Running 11 (18m ago) 35d rabbitmq-cluster-operator-86d6f94585-p4qkr 1/1 Running 11 (18m ago) 35d standby-replication-operator-69dc6689d5-qhggd 1/1 Running 8 (18m ago) 35d tanzu-rabbitmq-server-0 1/1 Running 1 (18m ago) 3d15h tanzu-rabbitmq-server-1 1/1 Running 1 (18m ago) 3d15h tanzu-rabbitmq-server-2 1/1 Running 1 (18m ago) 3d15h
2 months agoIf Customer purchases a license to VMware Tanzu RabbitMQ, Customer may use the Software for up to the number of Cores for which Customer has paid the applicable license fees, which can be deployed as ... VMware Tanzu RabbitMQ for Kubernetes ...
VMware Tanzu GemFire usage is measured by counting the number of Cores that are running
GemFire Locators and GemFire Servers. Total usage is determined by counting all Cores across
all environments running the VMware Tanzu GemFire Softwar
2 months ago
1 month agoOpen ⟶ Waiting on R&D2023-05-11T15:12:37.31+0000 [STG/0] OUT -----> Ruby Buildpack version 1.8.50The Ruby Buildpack used is 1.8.50, which is the one included with TAS 2.11.14. I can see on Github that Ruby 3.1.0 is not included in this buildpack:
2023-05-11T15:12:37.31+0000 [STG/0] OUT -----> Supplying Ruby
2023-05-11T15:12:37.33+0000 [STG/0] OUT -----> Installing bundler 2.3.4
2023-05-11T15:12:37.33+0000 [STG/0] OUT Copy [/tmp/buildpacks/c0c5c28629da5a963e748b27978689d0/dependencies/1c6e2232b3a154d37a9e088ff0a0d4a9/bundler_2.3.4_linux_noarch_any-stack_c0bedb3e.tgz]
2023-05-11T15:12:37.73+0000 [STG/0] OUT [31;1m**ERROR**[0m Unable to determine ruby: Unable to determine ruby version: Running ruby: No Matching versions, ruby ~> 3.1.0 not found in this buildpack
2023-05-11T15:12:37.75+0000 [STG/0] ERR Failed to compile droplet: Failed to run all supply scripts: exit status 15
2023-05-11T15:12:37.76+0000 [STG/0] OUT Exit status 223
ruby '~> 3.1.0'https://github.com/pivotal-cf/rabbit-example-app/blob/878fb0362c948988f766f3153d342c94b60bf3e7/Gemfile
1 month agoScan results for: image ***/rabbitmq-vmware:
+----------------+----------+------+---------+---------+-------------------------------------+------------+------------+----------------------------------------------------+-------------------+
2023-05-16T07:12:58.1389505Z | CVE | SEVERITY | CVSS | PACKAGE | VERSION | STATUS | PUBLISHED | DISCOVERED | DESCRIPTION | TRIGGERED FAILURE |
2023-05-16T07:12:58.1390523Z +----------------+----------+------+---------+---------+-------------------------------------+------------+------------+----------------------------------------------------+-------------------+
2023-05-16T07:12:58.1391530Z | CVE-2021-3177 | critical | 9.80 | python | 3.7.5 | | > 2 years | < 1 hour | Python 3.x through 3.9.1 has a buffer overflow | Yes |
1 month agosha256:72a3fc9c410791f32568c02f3b4ef4d3e3ca128fe71689f3dcff7b413c71e40frabbitmqadmin CLI, which is useful for gathering information about RabbitMQ
1 month ago
1 month ago
1 month ago2023-05-17 04:39:22.362020-04:00 [info] <0.2456.0> Supervisor {<0.2456.0>,rabbit_connection_sup}: child helper_sup started (<0.2457.0>): {rabbit_connection_helper_sup,start_link,[]}
2023-05-17 04:39:22.362226-04:00 [info] <0.2456.0> Supervisor {<0.2456.0>,rabbit_connection_sup}: child reader started (<0.2458.0>): {rabbit_reader,start_link,[<0.2457.0>,{acceptor,{127,0,0,1},5672}]}
2023-05-17 04:39:22.362489-04:00 [info] <0.2458.0> accepting AMQP connection <0.2458.0> (127.0.0.1:57770 -> 127.0.0.1:5672)
2023-05-17 04:39:22.363108-04:00 [debug] <0.2458.0> Raw client connection hostname during authN phase: {127,0,0,1}
2023-05-17 04:39:22.363198-04:00 [debug] <0.2458.0> Resolved client hostname during authN phase: 127.0.0.1
2023-05-17 04:39:22.363255-04:00 [debug] <0.2458.0> User 'gpadmin' authenticated successfully by backend rabbit_auth_backend_internal
2023-05-17 04:39:22.363554-04:00 [info] <0.2457.0> Supervisor {<0.2457.0>,rabbit_connection_helper_sup}: child collector started (<0.2459.0>): {rabbit_queue_collector,start_link,[<<"127.0.0.1:57770 -> 127.0.0.1:5672">>]}
2023-05-17 04:39:22.363684-04:00 [info] <0.2457.0> Supervisor {<0.2457.0>,rabbit_connection_helper_sup}: child heartbeat_sender started (<0.2460.0>): {rabbit_heartbeat,start_heartbeat_sender,[#Port<0.226>,10,#Fun<rabbit_reader.4.28767156>,{heartbeat_sender,<<"127.0.0.1:57770 -> 127.0.0.1:5672">>}]}
2023-05-17 04:39:22.363793-04:00 [info] <0.2457.0> Supervisor {<0.2457.0>,rabbit_connection_helper_sup}: child heartbeat_receiver started (<0.2461.0>): {rabbit_heartbeat,start_heartbeat_receiver,[#Port<0.226>,10,#Fun<rabbit_reader.5.28767156>,{heartbeat_receiver,<<"127.0.0.1:57770 -> 127.0.0.1:5672">>}]}
2023-05-17 04:39:22.364018-04:00 [info] <0.2457.0> Supervisor {<0.2457.0>,rabbit_connection_helper_sup}: child channel_sup_sup started (<0.2462.0>): {rabbit_channel_sup_sup,start_link,[]}
2023-05-17 04:39:22.364161-04:00 [info] <0.2458.0> connection <0.2458.0> (127.0.0.1:57770 -> 127.0.0.1:5672): user 'gpadmin' authenticated and granted access to vhost 'vhost_gpss'
2023-05-17 04:39:22.364894-04:00 [info] <0.2464.0> Supervisor {<0.2464.0>,tls_dyn_connection_sup}: child sender started (<0.2465.0>): {tls_sender,start_link,[]}
2023-05-17 04:39:22.365040-04:00 [info] <0.2464.0> Supervisor {<0.2464.0>,tls_dyn_connection_sup}: child receiver started (<0.2466.0>): {ssl_gen_statem,start_link,[server,<0.2465.0>,"localhost",5671,#Port<0.227>,{#{key => undefined,cert => undefined,signature_algs_cert => undefined,eccs => {elliptic_curves,[{1,3,132,0,35},{1,3,132,0,34},{1,3,132,0,10},{1,2,840,10045,3,1,7}]},certfile => <<"/etc/rabbitmq/ssl/cert_new.pem">>,max_handshake_size => 262144,keyfile => <<"/etc/rabbitmq/ssl/key_new.pem">>,dhfile => undefined,depth => 10,certificate_authorities => undefined,dh => undefined,use_ticket => undefined,anti_replay => undefined,supported_groups => {supported_groups,[secp256r1,secp384r1]},signature_algs => [{sha512,ecdsa},{sha512,rsa},{sha384,ecdsa},{sha384,rsa},{sha256,ecdsa},{sha256,rsa},{sha224,ecdsa},{sha224,rsa},{sha,ecdsa},{sha,rsa},{sha,dsa}],crl_check => false,ocsp_nonce => true,password => [],padding_check => true,customize_hostname_check => [],sni_fun => undefined,server_name_indication => undefined,psk_identity => undefined,hibernate_after => infinity,partial_chain => #Fun<ssl.11.30095513>,user_lookup_fun => undefined,cookie => true,ocsp_responder_certs => [],alpn_advertised_protocols => undefined,versions => [{3,3},{3,2},{3,1}],alpn_preferred_protocols => undefined,log_level => notice,cacerts => undefined,fail_if_no_peer_cert => true,verify => verify_peer,verify_fun => undefined,max_fragment_length => undefined,reuse_session => #Fun<ssl.13.30095513>,reuse_sessions => true,cacertfile => <<"/etc/rabbitmq/ssl/cacert_new.pem">>,honor_ecc_order => false,key_update_at => 388736063997,handshake => full,ocsp_stapling => false,fallback => undefined,ciphers => [<<"À,">>,<<"À0">>,<<"À
">>,<<"À¯">>,<<"À$">>,<<"À(">>,<<"À+">>,<<"À/">>,<<"À¬">>,<<"À®">>,<<"À.">>,<<"À2">>,<<"À&">>,<<"À*">>,<<"À-">>,<<"À1">>,<<"À#">>,<<"À'">>,<<"À%">>,<<"À)">>,<<0,159>>,<<0,163>>,<<0,107>>,<<0,106>>,<<0,158>>,<<0,162>>,<<0,103>>,<<0,64>>,<<"À\n">>,<<192,20>>,<<192,5>>,<<192,15>>,<<"À\t">>,<<192,19>>,<<192,4>>,<<192,14>>,<<0,57>>,<<0,56>>,<<0,51>>,<<0,50>>],client_renegotiation => true,protocol => tls,sni_hosts => [],next_protocols_advertised => undefined,erl_dist => false,secure_renegotiate => true,middlebox_comp_mode => true,next_protocol_selector => undefined,session_tickets => disabled,beast_mitigation => one_n_minus_one,honor_cipher_order => false,srp_identity => undefined,crl_cache => {ssl_crl_cache,{internal,[]}},keep_secrets => false,renegotiate_at => 268435456,early_data => undefined},{socket_options,binary,raw,0,0,false},[{option_tracker,<0.733.0>},{session_tickets_tracker,disabled},{session_id_tracker,<0.734.0>}]},<0.739.0>,{gen_tcp,tcp,tcp_closed,tcp_error,tcp_passive}]}
2023-05-17 04:39:22.365938-04:00 [info] <0.2467.0> Supervisor {<0.2467.0>,rabbit_connection_sup}: child helper_sup started (<0.2468.0>): {rabbit_connection_helper_sup,start_link,[]}
2023-05-17 04:39:22.366063-04:00 [info] <0.2467.0> Supervisor {<0.2467.0>,rabbit_connection_sup}: child reader started (<0.2469.0>): {rabbit_reader,start_link,[<0.2468.0>,{acceptor,{127,0,0,1},5671}]}
2023-05-17 04:39:22.370675-04:00 [notice] <0.2466.0> TLS server: In state certify at tls_dtls_connection.erl:314 generated SERVER ALERT: Fatal - Handshake Failure
2023-05-17 04:39:22.370675-04:00 [notice] <0.2466.0> - no_client_certificate_provided
2023-05-17 04:39:22.371541-04:00 [info] <0.2458.0> closing AMQP connection <0.2458.0> (127.0.0.1:57770 -> 127.0.0.1:5672, vhost: 'vhost_gpss', user: 'gpadmin')
2023-05-17 04:39:22.371891-04:00 [debug] <0.2471.0> Closing all channels from connection '127.0.0.1:57770 -> 127.0.0.1:5672' because it has been closedOpen ⟶ Waiting on R&Dgatherer, while the result never comes because the worker_pool_worker…gatherer, while the result never comes because the worker_pool_worker…
28 days ago
28 days ago
28 days agorabbitmqctl eval 'rabbit_vhost:msg_store_dir_path(<<"/">>).'
28 days agorabbitmqctl eval '<<Num:128>> = erlang:md5(<<"queue", <<"VHOST_NAME">>/binary, <<"QUEUE_NAME">>/binary>>), rabbit_misc:format("~.36B", [Num]).'
13 days agoactivemqconfigdefaults: &eventMqConfigBase brokerUrl: failover:(ssl://rce-activemq:61616?wireFormat.maxInactivityDurationInitalDelay=60000)?jms.prefetchPolicy.queuePrefetch=1 username: rce password: rce_jms_password ignoreCert: true queue: sddc-metrics useTopic: false acksAsync: false createConnectionOnStartup: true maximumActiveSessionPerConnection: 50 maxConnections: 50 maxThreadPoolSize: 50
public ConnectionFactory createMqConnectionFactory() {
ActiveMQConnectionFactory connectionFactory;
if (ignoreCert) {
log.info("Creating ActiveMqConnectionFactory without certificate verification");
connectionFactory = new ActiveMqSslConnectionFactoryNoVerify(username, password, brokerUrl);
} else {
log.info("Creating ActiveMqConnectionFactory with certificate validation.");
connectionFactory = new ActiveMQConnectionFactory(username, password, brokerUrl);
}
log.debug("Connection parameters acksAsync: {} producerWindowSize: {} maxThreadPoolSize: {} "
+ "maximumActiveSessionPerConnection: {} maxConnections: {}", acksAsync, producerWindowSize,
maxThreadPoolSize, maximumActiveSessionPerConnection, maxConnections);
connectionFactory.setNonBlockingRedelivery(true);
connectionFactory.setSendAcksAsync(acksAsync);
if (acksAsync) {
connectionFactory.setProducerWindowSize(producerWindowSize);
connectionFactory.setMaxThreadPoolSize(maxThreadPoolSize);
}
RedeliveryPolicy redeliveryPolicy = new RedeliveryPolicy();
redeliveryPolicy.setInitialRedeliveryDelay(10000L);
redeliveryPolicy.setMaximumRedeliveries(-1);
connectionFactory.setRedeliveryPolicy(redeliveryPolicy);
PooledConnectionFactory pooledConnectionFactory = new PooledConnectionFactory();
pooledConnectionFactory.setCreateConnectionOnStartup(createConnectionOnStartup);
pooledConnectionFactory.setMaximumActiveSessionPerConnection(maximumActiveSessionPerConnection);
pooledConnectionFactory.setMaxConnections(maxConnections);
pooledConnectionFactory.setConnectionFactory(connectionFactory);
return pooledConnectionFactory;
}
/**
* Returns a JmsTemplate for active Mq.
*
* @return JmsTemplate.
*/
public JmsTemplate createJmsTemplate() {
return this.createJmsTemplate(createMqConnectionFactory());
}
protected JmsTemplate createJmsTemplate(ConnectionFactory connectionFactory) {
JmsTemplate template = new JmsTemplate(connectionFactory);
if (isUseTopic()) {
template.setPubSubDomain(true);
} else {
template.setDefaultDestinationName(queue);
}
return template;
}
7:59 PM